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This paper offers a humorous parody of the vay in 
vhich evaluation teams often enter clinical institutions to do 
program evaluat;Lon studies. The parody emphasizes the over-use of 
sophisticated tecjtinigues and methodologies; as veil as expensive 
equipment. The authors then define evaluation as "determining the 
degree to vhich a program is meeting its objectives, the problems it 
is encountering and t>e side effects it is creating." They contend 
that in order to evaluate effectively, one must consider the context 
of the program. Program objectives should be stated in terms of 
structure, process and outcome, all of vhich are further defined. 
Sociodemo graphic data is generally available about th^ specif:^'q 
populations served and should also be utilized. Outcome eyalui4:iop is 
seen as the most difficult and costly type of evaluationv but of 
extreme importance. Outcomes should be defined and explicated. 
Simplifying the terminology and process of evaluation is essential 
for improving human service delivery. (NG) 
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Imagine it! A convoy, led by the "program evaluation 
van", rolls through the gate of "Mending Minds" Psychiatric 
Hospital and parks in a circular formation on the ninth hole 
of the hospital's golf course. Members of the crew gingerly 
unload highly volatile tanks of G.A.S.,-"" ATGON,^ and SCL-90.^ 
Other crewmen rapidly p)lug the van into a power source as 
technicians activate the van's System 3 computer. It's 
capabilities are augmented by virtual memory and dynamic 
reallocation. The process of unloading complex and sophisti- 
cated mental health program evaluation gear continues as the 
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^PATS team prepares its PEP procedures and program auditors 

5 

conduct a PASS Review. 

Several large hoses are removed from their racks and 

connected to recepticles in nearby wards. The equipment begins 

to whir and grind as vacuum suction pulls in some of the ward's 

atmosphere. Immediately, complex analyses are performed to 
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produce a score on the Ward Atmosphere Scale. Highly sensitive 
sensors measure organization climate^ and levels of staff empathy 

Quietly and efficiently "Mending Mind's" patients are led 
to the entrance of interconnected semi-trailers in which they 
pass through an optical scan device (laser based and similar to 
those new-fangled checkout counters in the supermarket) which 
quickly reads their identity from the small lines printed upon 
their wrist bracelets. They are, then, seated in comfortable arm 
chairs attached to a conveyor belt (just like in the Haunted 
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Mansion at Florida's Disney World). Gradually, they laove 
through a series of stations at which their physical health 
and mental status are examined, As they exit, results of 
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this multiphasic screen, measures of their ftinctional status, 
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adaptive behavior skills, level of psychopathology and an 
instant diagnosis are miraculously produced. 

Simultaneously, the staff as they munch on "Evalutreats",-'" 
are entertained by *'Reddy Evaluator", an electronic marvel on 
the side of the van who preaches the gospel of program evaluation 
and presents the most recent dymaxion (20-sided) projection''"''" of 
evaluative levels and utility for decision -making. 

Now wait a minute! This is getting terribly absurd 1 What 

does all of this mean? G.A.S., for instance, is an acronym used 
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with reference to the Global Assessment S^cale*^ or as a generic 
label applied to a number of goal attainment £caling strategies 
including Kiersuk's "Goal Attainment Scaling, Wilson* s ATGON 
(Automated Tri-Informant Goal Oriented Progress Note) and 
Eval-U-Treat developed by Benedict. Rather than being a highly 
volatile substance, SCL-90 is a 90 item symptom check list. PAT 
(Psychiatric Audit Team) members perform the JCAH's Performance 
Evaluation Procedure for Auditing and Improving Patient Care 
while other teams utilize Wolfensberger ' s Program Analysis 
of Service Systems methodology. Certainly, these are all appro- 
priate and well-developed methodologies. Likewise, examination 
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of ward atmosphere and its consequences or the impact of 

organization climate on. treatment outcome are important elements 

of program evaluation. Cubistic models of evaluation strategy, 

iQ 14 

including those published by Schwab and Warheit, Bell, and 
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Atkisson and colleagues, also have their value. However, are 
all of these necessary, desirable or, even more important, do 
they remove the basis of responsibility for decision-making 
from the clinical manager and evaluation from the context of 
the service delivery system? '*Tlie inability of a majority of... 
directors and administrators to utilize these complex computer- 
based evaluation systems does not negate, however, the importance 
of 'evaluation' ."^^ 

Trappings, jargon and inordinate concern with issues more 
appropriate to clinical research, distract from our view of 
evaluation as "determining the degree to which a program is 
meeting its objectives, the problems it is encountering and 
the side effects it is creating. "■'■^ "In practice, evaluation 
services usually go beyond this definition, to concern themselves 
with the need for the program, its scope in relation to the needs, 
and the internal processes of program ftinctioning . This extension 
is necessitated by the fact that program effectiveness often can 
be discerned only through examination of the program's context 
and its operational processes."''"^ 

"Evaluations exist (or perhaps only should exist) to 
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facilitate intelligent decision-making. ''•'"^ In our view, 
evalxiation is a tool to allow the clinical manager to make 
rationale data-based decisions and to alert him to situations 
in which management intervention may be necessary. "Evaluation 
may provide data which will reduce uncertainty as to what's really 
happening inside the program and begin to clarify the pluses 
and minuses of various decisions. In a way it will allow us to 
see the values and preferences of the decision-makers more clearly. "2 
From this perspective, evalxiation can be done using less sophisti- 
cated techniques, less technology and with minimal allocation of 
resources. All of the trappings and expensive equipment mentioned 
above may be superfluous to the tasks at hand. 

This is not to say that those techniques and methodologies 
explicitly or iirplicitly referenced in our parody are not useful, 
practical or reliable. Rather, they are, with due recognition 
of both their assets and deficits, respectable components of the 
evaluative technology. It is the obfuscation created by these 
acronyms and the technical complexity with which they are presented 
which detract from their ready implementation. They may be 
applicable to many issues that clinical managers may wish to 
address and can be utilized without reference to the aura of 
reverence or complexity which surrounds them. In point of fact, 
we encourage evaluators to utilize existing instrumentation and 
technology rather than to expend valuable resources on what is 
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often duplicative. 

There are multiple levels of evaluation. We can do 
evaluation at some level even without technical expertise, 
computer access or high resource expenditure. We can accomplish 
more with the availability of technical experts. 

Availability of technical experts with the infusion of 
state of the art technology and almost unlimited resources can 
often achieve evaluative "nirvana" (as a state of self-delusion) . 
However, the lack of technocrats, computer access, and unlimited 
resources IS NO EXCUSE FOR FAILURE TO EVALUATE. 

Basic to the capability to evaluate is a context in which 

program objectives , either explicit or implicit, exist and are 

understood. These can, and should, be stated in terms of structure, 
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process and outcome. We advocate a rather simple "verb-noim" 
approach to setting clear, concise, and easily understood goals, 
for example: "reduce length of stay/' These goals then can be 
operationalized by first determining the present level or baseline 
(i.e., current length of stay is 30 days), determining a target, 
neither too easily reached nor too difficult to attain (i.e., 
reduce length of stay by 4 days) and specifying a deadline (i.e., 
within 6 months). Remember, these objectives are not cast in 
bronze, they can and should be revised with experience. 

Objectives addressing structural program issues, such as 
adequacy of physical plant, of staff /patient ratio and/or staff 
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qiialifications as related to program needs, while the subject 
of much external review in the form of inspection for compliance 
with standards, can be the basis for rigorous periodic self 
evaJIuation. The clinical manager can compare the current program 
environment and staffing against his objectives or JCAH (Joint 
Commission on the Accreditation of Hospitals) standards relevant 
to his particular type of facility, and initiate intervention 
where appropriate. Coimnon structural goals such as: 

1. Decrease patient/direct service staff ratio, 

2. Hiamanize wards, and 

3. Develop outreach clinics, 

can be specified in ways which will make them measurable and give 
the clinical manager real information concerning program structure. 
The objectives written from the above might be: 

1. Decrease the existing patient/psychologist ratio (100 
to 1) to 50 to 1 by the end of the current fiscal year, 

2. Hang drapes meeting 1974 - federal fire resistant 
regulations in all residential living units within 3 months, and, 

3. Open a store front satellite clinic within one year 
in the identified high risk area of "Freud's Station." 

Within the realm of process, it is possible in programs 
with limited access to technocrats and resources to make use of • 
the already available information base at the service delivery 
level. Systematic examination of these data or their recombination 
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into indices or ratios can provide an insight into attainment 

of process objectives. 

Process evaluation looks at the 
operations of the program, the practices 
within it, (the patterns which develop), 
and the ways that program approaches and 
. interacts with the mental health system. . . 
It is useful as a qxiality measure when, 
process results are compared against 
baseline data, norms, standards and 
realistically determined objectives. 

Since in most cases process objectives 
are indirect measures of quality, why look 
at proceOiBl Process evaluation is immedi- 
ately attainable. It relies on a basic 
data set... which may already exist or can, 
with little effort (and additional cost) 
be developed within a program. Measurable 
and observable process objectives can be 
set for areas and population groups to be 
served, modalities of service to be used, 
equity and availability of services, and 
continuity of care within the system based 
on needs assessment, census data and base-* 
line information on current operations . 

For example, a goal of "reducing injury producing 

incidents" has been set. This has been operationalized as 

"reduce by 507o the frequency of injury (average of 8 per month) 

to residents over the next 6 months." This objective can then 

be monitored by using incident reports which are often completed 

when an injury causing event occurs. Typically, one copy of 

such a report is sent to a review committee and one is filed in 

the involved individual's record. These incidents are then 

reviewed on a case-by-case basis, which of course is necessary. 

Rarely, however, does the clinical manager keep a continuous 
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record of these incidents to determine if any patterns emerge 
as to time, place, type of incident or individuals involved. 
Yet, this is exactly the Information he needs to determine 
the achievement of his stated objective. 

Similar uses can be made of aggregate morbidity and 
mortality information, medication profiles, or, even examination 
of changing patterns of utilization of isolation rooms, restraints 
emergency ambulance services, and intramuscular injections 
(especially when used for behavior management) . Longitudinal 
examination of the entire range of treatment modalities provided 
can also illuminate the existing operational processes. Still 
within the context of readily accessible data sources is the 
treatment documentation yielding information about the process 
of patient care. A system of explicit objectives, or criteria, 
is necessary to focus the review of treatment docximentation and 
to select information elements to be abstracted, such as "each 
therapist will develop at least 3 observable behavioral 
objectives for each of his patients within one month." Simple 
examination to ascertain whether documentation is completed 
within prespecified time constraints, whether indicated risks 
are managed in accordance with criteria for risk management, 
whether services are provided within a temporally appropriate 
sequence, and whether adequate justification and documentation 
exists for continued maintenance at the current level of care is 
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possible based upon accepted criteria. A methodology for such 

review using existing treatment documentation, variously termed 
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psychiatric *audit or medical care evaluation studies has been 
proposed by the JCAH under the title Performance Evaluation 
Procedure . 

In ambulatory settings aggregation of data concerning 
"no shows" and "service dropouts" in terms of frequency of 
occurrence, and a time of day and day of week format can reveal 
patterns of utilization and accessibility. Against a goal such 
as "improve temporal accessibility," these data can be used by 
the clinical manager to schedule services and staff time at high 
utilization periods, and conversely, to schedule administrative 
fimctions such as staff meetings during periods of low utiliza- 
tion. Thus objectives of "move staff meetings to Monday morning 
by March" and "open the outpatient clinic three nights a week 
by April" might be established. 

Simple comparisons of soclodemographic data concerning 
the population served and the general population can provide a 
wealth of information on who the program is reaching and how 
well it is doing in terms of its own assumptions and/or community 
expectations of who it should be serving. A readily available 
source for these data are the publications of the Bureau of the 
Census as well as the entire National Institute of Mental Health 
Series C publications (including titles such as "A Model for 

11 
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Estimating Mental Health Needs Using 1970 Census Socioeconomic 

Data") which provide a working model for the clinical manager 

on not only evaluation, but also topics such as accounting^ 

cost finding and information systems. As technocrats and 

resources become available, sociodemographic coiapafisons can be 

made not only between the population served and the general 

population, but also between the population served and the 

population-at-risk as defined using some sociotechnologic 

approximation of prevalence. 

Outcome evaluation is generally considered the highest 

level of evaluation and therefore it is assumed to require the 

greatest infusion of resources for its support. A major reason 

for its cost is that it is often geared toward assessing the 

individual and attempts to measure change in condition over 

time. Measuring change of the htiman psychopatho logic, symptomatic, 

or funttional condition generally requires standardized instrtnnents 

or techniques. These, in turn, require training to administer,* 

ability to analyze, and at least two administrations (pre and 

post) for purposes of interpretation. (Ciarlo, has proposed 
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a ''post^' only methodology which may be effective as well) . 

These techniques generally key on multiple items or 
specific individualized goals. The goal attainirant scaling 
techniques pervious ly mentioned all rely, in one form or another, 
on a simple process of a therapist setting an observable objective 
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arid monitoring progress toward /aat objective on a prespecified 
continutim. However, it is possible to look at a limited set 
of conditions or goals to whir' '""^^ " togram is specifically 
directed. For instance, u retardation program a 

pro toii^ww liairi the residents:" FolloWlug 

review of baseline performance data, this goal can be 
operationalized by stating, "75% of the residents will toilet 
independently after completing the six week training program. " 
In certain settings, such as a chronic geriatric lanit, a 
maintenance goal may be established which targets maintenance of 
a fimctional state as a successful outcome. An example might 
be "maintain present level of ftmctioning for 977o of patients 
over the next six months." 

The effects of utilization review and the development 
of PSRO's have already led to a more well-defined data base in 
inpatient programs which may be useful for outcome evaluation . 
Within the utilization review process , it is necessary to 
establish criteria with which to determine whether admission and 
continued stay are appropriate. It is also possible to measxire 
symptom relief, improvement or change in ftinction based on those 
same criteria. At this level of evaluation instruments such as 
the Global Assessment Scale or the SCL-90 become useful in 
measuring progress along pre-determined and standardized items. 

From the foregoing examples, by no means the universe 
. , '■,'•«..•■■'■■.. 
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of techniques available to the clinical manager we hope that 
we have shown that it is in fact possible to conduct evaluation, 
without becoming mired in the jargon of the field or expending 
an inordinate proportion of the service delivery systems 

resources 

We, as evaluators, do ourselves and our clients a 
disservice by allowing technical shorthand to infiltrate our 
communications to the detriment of our concern for rational 
data-based management toward the goal of effective and efficient 
human service delivery. 
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